Musings on Deep Learning: Properties of SGD

We ruminate with a mix of theory and experiments on the optimization and generalization properties of deep convolutional networks trained with Stochastic Gradient Descent in classification tasks. A present perceived puzzle is that deep networks show good predictive performance when overparametrization relative to the number of training data suggests overfitting. We dream an explanation of these empirical results in terms of the following new results on SGD: 1. SGD concentrates in probability like the classical Langevin equation – on large volume, “flat” minima, selecting flat minimizers which are with very high probability also global minimizers. 2. Minimization by GD or SGD on flat minima can be approximated well by minimization on a linear funcion of the weights, suggesting pseudoinverse solutions. 3. Pseudoinverse solutions are known to be intrinsically regularized with a regularization parameter λ which decreases as 1 T where T is the number of iterations. This can qualitatively explain all the generalization properties empirically observed for deep networks. 4. GD and SGD are connected closely to robust optimization. This provides an alternative way to show that GD and SGD perform implicit regularization. These results explain the puzzling findings about fitting randomly labeled data while performing well on natural labeled data. They also explain while overparametrization does not result in overfitting. Quantitative, non-vacuous bounds are still missing, as it has almost always been the case for most practical applications of machine learning. This is version 3, which differs from version 2 only in the title. The first version was released on 04/04/2017 at https://dspace.mit.edu/handle/1721.1/107841.